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. " A study examined the retention of mathematical and 

reading comprehension skills for students enrolled in a 
learning-in- work environment (experience-based career education) and 
a traditional classroom learning environment « Academic achievement 
was measured using a twelve-month longitudinal design. Student 
performance in each environment was evaluated on the Comprehensive 
Test of Basic Skills administered at beginning and end of the junior 
year and beginning of the senior year. The learning interval was 
designated as the time between pre- and post-testing r and the 
retention interval as the time between post- and follow-up testing. 
The results indicated (1) differences in two areas-^in reading skills 
versus math skills and in traditional learning environments versus 
learning-in- work environments; (2) math congnitive concepts causally 
related but net in the hierarchical order predicted by Bloom^s 
taxonomy: (3) different causal processes in the acquisition of math 
ccncepts, depending u'^on placement in one of the two learning 
..environments: (U) significant relationships between students* 
cognitive style and their math performance: and (5) moderate 
relationships between students • perceptions of complexity of learning 
environment and their math performance. An interference/assimilation 
irodel was proposed to interpret findings, (Technical information and 
appendixes are available as CE 027 94 3,) (YLB) 
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FOREWORD 



Learning-in-work is an integral part of Experienced-Based 
Career Education Programs^ internships, cooperative and work- 
experience programs, and on-the-job components of vocational 
education. Since the early 1970s, there has been a movement in 
education to expand the education opportunities of all students 
to include "real world" learning experiences as part of the total 
educational experience. In an attempt to investigate the rela- 
tionships of learning and work, the National Center for Research 
in Vocational Education has initiated a programmatic effort to 
conduct basic research of the phenomena. This study, supported 
by the National Institute of Education, reports the findings of a 
two-year exploratory examination of student retention of 
mathematical and reading skills resulting from student enrollment 
in a learning-in-work and in a traditional learning environment. 

Suggestions for extending the first year's results were 
provided by David P. Ausubel, Distinguished Professor Emeritus, 
Ph.D. Program in Educational Psychology, Graduate School*, City 
University of New York; Henry C. Ellis, Chairman, Department of 
Psychology, University of New Mexico; and Benton J. Underwood, 
Stanley G. Harris Professor of Social Science, Northwestern 
University. These scholars were commissioned to provide a 
perspective for investigating the psychological and pedagogical 
implications of learning and forgetting patterns in learning-in- 
work and traditional environments through extensions of their 
research and learning theories. While their suggestions are 
incorporated in the executive summary, their complete papers 
appear in the technical report. 

Appreciation is extended to the Anoka-Hennepin School 
District No. 11, Anoka, Minnesota, for their cooperation and 
participation in the study. Don Anderson, Director of the 
Experience-Based Career Educaction Program, and Roger Giroux, 
Director of Research and Evaluation, were instrumental in 
providing support for the research staff in their investigation. 

Technical advice for the research effort was provided by 
Harold M. Schroder, Professor and Department Chairman of the 
College of Business Administration at the University of South 
Florida. His scholarly research in the area of complexity 
training and development of individuals was especially useful to 
the research staff • He is also recognized for his critique of 
the report. 

vii 
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ABSTRACT 



The study examined the retention of mathematical and reading 
comprehensicTi sk'.lls for students enrolled in a learning-in-work 
environment (Experience-Based Career Education) and a traditional 
classroom learning environment on a measure of academic achieve- 
ment in a twelve-month longitudinal design* Student performance 
(n=27) in each environment was evaluated, using the Comprehensive 
Tests of Basic Skills/ which was administered at the beginning 
and end of their junior year and at the beginning of their senior 
year. Thus, the learning interval was designated as the time 
between pre- and post-testing, and the retention interval, the 
time between post- and follow-up testing. The results indicated: 
(1) that students demonstrated significantly different learn- 
ing/retention functions for math skills depending upon parti- 
cipation in one of the two learning environments; the learning 
and retention patterns of the students in the traditional 
learning environment conformed to the predictions from previous 
research — demonstrating increased performance during the 
instructional or learning interval and decreased performance 
during the summer break or retention interval, while the patterns 
of students in the learning-in-work environment deviated from 
predicted patterns — demonstrating decreased performance during 
the learning interval and increased performance during the 
retention interval; (2) that students in both learning 
environments demonstrated generally equivalent performance 
patterns for reading comprehension , skills that increased linearly 
over both the learning and retention intervals; (3) that math 
cognitive concepts were causally related: for the traditional 
learning students, the causally-learned concepts did partially 
support the hierarchical order theorized by Bloom's taxonomy, 
while for the learning-in-work students, the causally-learned 
concepts did not support the hierarchical order theorized by 
Bloom^s taxonomy; (4) that students who demonstrated greater 
cognitive complexity performed significantly higher on their 
acquisition of math skills; and (5) that students who perceived 
greater complexity in their learning environment generally 
performed lower on their acquisition of math skills. An 
interference/assimilation model was proposed to interpret the 
findings. 
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INTRODUCTION 



This executive summary is a digest of information resulting 
from a preliminary two-year effort to investigate the patterns of 
learning and retention of basic skills in learning-in-work and 
traditional learning environments. In the spirit of a digest, 
salient points from the two technical reports (Crowe and Harvey 
1979 r 1980) are highlighted and discussed. The primary purpose 
is to communicate the findings, provide interpretations, and set 
forth a research agenda and recommendations in such a way that 
practitioners, decisionmakers, and policy analysts can use the 
information as a basis for designing, operating, and evaluating 
learning-in-work programs. Such topics as sampling and design 
considerations, instrument development and validation, analytical 
strategies, and statistical manipulations are briefly overviewed. 
For a thorough treatment of these topics, readers are referred to 
the technical reports. 

Historically, learning- in-work programs have been a part of 
the tradition of alternative educational opportunities provided 
for young people. Our use of the phrase, however, refers to the 
more recently developed systems such as Experience-Based Career 
Education, Executive High School Internships Programs, and Career 
Intern Programs that were initially designed to meet the 
increasing demands for more relevant education and for career 
education. A common component of these programs is that youth 
leai>n experientially in work environments under the guidance of 
mentors from the work organization and the supervision of school 
personnelo Parenthetically, it is our view that, consciously or 
not, educators realized that the environment encountered after 
schooling was becoming more complex, while learning in the 
classroom environment was continuing to stress the invariant 
knowledge, rules, and processes for dealing with known and 
recurring problems. Since this mis-match was not compatible with 
the increased complexity required for problem solving and 
productivity in sophisticated technological work environments, 
more attention was directed to learning environments that would 
provide transitive contexts for. youth entering the workplace. 

Current legislative considerations for the youth initiatives' 
emphasize employability development and programs designed to 
assist youth in acquiring employability competencies, not the 
least of which is a mastery of basic skills.. Furth^ermore , not 
only do existing or proposed employability' development programs 
include an experiential learning component in the workplace, but 
they also assume that learning can occur experientially, learning 
that is more productive than traditional .classroom learning. 
While this position is appealing, it is a fact that experiential 
learning can also be accomplished within the classroom. Thus, 



when the underlying dimensions of learning-in-work environments 
were investigated, the following questions emerged. To what 
extent does learning exper ieiitially in a work environment enhance 
or suppress Xeafning employability skills? More specifically, as 
related to tl:is study, what are the patterns of learning and 
retention of the basic skills of mathematics and reading compre- 
hension learned exper ientially in a work environment and learned 
traditionally in a classroom? Are the cognitive processing 
functions of learning and retention different for the two 
environments? 

This investigation, which is only exploratory, is an initial 
effort to fill the gap between what is known from basic research 
(in laboratory settings with experimental designs) regarding cog- 
nitive processing and basic skill development and what is observ- 
ed in naturalistic settings regarding the learning and retention 
of basic skills. Overall, it is designed as an heuristic study 
in that its purpose is to generate ideas, to encourage further 
inquiry of this type, and to open up new lines of investigation. 

In the light of this overall purpose, the remainder of the 
executive summary is organized as follows: 

o The Problem 

The line of inquiry and the heuristic model used 
for investigating student cognitive - processes , and 
the research questions are discussed. 

o Me t hodological Considerations 

The methodology is briefly described, including the 
research design, the instruments, and the data 
base» 

o Results 

The findings related to the research questions are 
presented and briefly discussed. 

o Interpretation 

The results are anchored to the theories of 
learning and retention. 

o Discussion and Recommendations 

The research findings and interpretations are 
discussed and related to implications for continuing 
this line of inquiry. 



THE PROBLEM 



■ Overview 

The growth of learning-in-work programs expands student 
learning opportunities to Include workplace environments, and 
provides young people the option of choosing one learning 
environment over another (Crowe and Adams 1979). Allowing such a 
choice stems from a supposition that students in learning-in-work 
environments learn traditional subject-matter concepts as well as 
or better than those in the traditional classroom environment. 
Proponents of this theory believe that students in the learning- 
in-work environments are exploring real problems in the workplace 
that relate to subject matter disciplines, are learning under the 
guidance of a mentor who solves work-related problems, are having 
experiences that pair subject matter concepts and specific work 
tasks, are learning concepts in a concrete, "hands-on" manner, 
and presumably are forming cognitive structures for storing and 
retrieving concepts similar to those of the mentor at the 
workplace. This study is an initial investigation of these 
assumptions. Before proceeding with the theoretical considera- 
tions, a description of the learning-in-work program is 
provided. 



Learning- in-Work Program Description 

Briefly, the Experience-Based Career Education (EBCE) model 
is an academically oriented, community-based program. Students 
spend one day per week at the learning center with a learning 
coordinator, who supervitr^^s and directs their learning activi- 
ties, and spend four days at a community worksite under the 
guidance and supervision of a resource person, the worksite 
mentor. They work at three to twelve sites per school year, 
depending on their career interests and academic needs. The 
program offers twenty-eight EBCE courses that are all related to 
traditional subject-matter disciplines. Students choose EBCE 
courses according to their assessment of (1) career interests and 
aptitudes (e.g., EBCE courses and occupations are related through 
the Dictionary of Occupational Titles worker trait groups) and 
(2) academic needs and interests (e.g., EBCE courses and 
traditional subject-matter concepts are related through an 
instructional matrix that sets the parameters for designing the 
learning activities) . 

Under the EBCE model, academic development is accomplished 
through the use of an activity sheet designed to guide students 
through a series of learning activities that relate subject- 
matter concepts and career objectives to experiences at the work- 
site. The activity sheets detail student tasks, estimate the 



time required to complete the activity, and furnish the basis for 
measuring progress. Students may be completing up to five or six 
activity sheets at any given time. Since activity-sheet assign- 
ments_are .expected to be . completed, at the worksiter. there are no 
homework expectations for the students. After concluding work as 
outlined on the sictivity sheet, the students and coordinators (on 
a one-to-one basis) jointly evaluate the assignments. Students 
are given the opportunity to do additional work if they desire a 
better evaluation. 



Research Related to Learning and Retention of 
Basic Skills in Alternative Learning Environments 

The EBCE approach would seem to offer the benefits of fre- 
quent, meaningful, and prompt feedback on task performance for 
students. Research in academic learning and retention (Ausubel 
1968, Boker 1974, La Porte and Voss 1975) would predict increased 
performance as a function of the above variables. Ausubel (1968) 
reported increased learning and retention of meaningful material 
as contrasted with rote-memorized material. Anderson and Biddle 
(1975) and Boker (1974) found a strong relationship between 
increased application (practice) of material and subsequent 
rccention. Similarly, La Porte and Voss (1975) demonstrated 
superior retention as a result of usage of the information and 
response-contingent performance feedback. 

While these factors are present to varying degrees within a 
classroom environment, we believe that the EBCE environment 
offers structurally-guided opportunities to encounter situations 
that emphasize these variables. For example, EBCE students 
complete tasks related to subject-matter concepts and complete 
them within the context of the workplace. They work with adults 
negotiating their work assignments and are required to meet the 
expectations of the learning coordinator, resource person, and 
coworkers employed by the organization. For us, these types of 
activities would seem to expand the number and variety of 
learning opportunities related to the above variables that 
learning- in-work students receive. 

As stated, we believe that the learning environment an 
individual chooses (EBCE vs. traditional classroom instruction 
approach) is important with respect to its effect on academic 
learning and retention; however, research has indicated that 
person-related variables such as cognitive style interact with 
the environment, suggesting that the difference in environment 
alone cannot account for all differences in learning and reten- 
tion. Davey (1976) concluded that the students' cognitive style, 
defined as stable preferences in individuals with respect to 
conceptual categorization and perceptual organization of the 
external environment, was a critical factor in maximizing 
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performance* Hunt (1975) argued for future research directed at 
identifying the interactive nature of the personal and environ- 
mental characteristics of behavior. Cronbach and Snow (1977), 
maintaining that the traditional method of classroom instruction 
was not optimal for all students, advocated consideration of the 
interaction of instructional methods with individual character- 
istics. Schroder, Driver, and Steurfert (1967) found that more 
complex persons were less influenced by the environment than less 
complex persons. In the same vein, Staszkiewicy (1977) indicated 
more cognitively complex studernts scored higher than less complex 
students in situations characterized by less teacher direction. 
Consequently, a closer examination of the interaction between 
person variables ( cognitive style) with learning environment 
(EBCE vs. traditional classroom instruction) in the present 
investigation seeks to examine the contributions of each with 
respect to academic learning and retention. 



Orientation to Investigating Learning and 
Retention of Basic Skills in the Two Environments 

Since past research using only a pre-test/post-test control 
group design has detected few statistical differences in 
students' achievements (Crowe and Adams 1979; Crowe and Walker 
1977), it was of interest in the present investigation to test 
the assumption that demonstrable changes in student performance 
may bccur sometime after students leave the program. Thus, a 
retention model design was proposed using repeated measures with 
the learning interval being the time students participated in the 
learning environment (nine months) and the retention interval, 
the summer recess (three months). In view of previous research 
on environmental contingencies as determinants of learning and 
retention (see Gagne 1978, for a review), it was predicted that 
students' exposure to different types of learning environments 
would affect performance on standardized tests of academic 
ability. The study examines performance in two learning 
environments longitudinally in order to detect changes in the 
retention of academic performance resulting from participation in 
one of the two environments during the learning interval. 

Studying work and its effect on individuals and their 
performance is in itself a complex phenomenon. However, it 
becomes even more complex when students are expected to learn 
subject matter at the same time that they are assimilating the 
working role. Therefore, in order to study the processes 
underlying learning in a work environit^ent, it becomes necessary 
to understand the student as both learner and worker, the 
dynamics of the environment, and the interaction of the student 
within the context of the work setting. 
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The environment in which subject matter is learned may shape, 
the way students perceive the meaningf ulness of the content and 
concepts associated with the discipline and therefore what they 
learn and remember. For example, it would seem to us that 
learning math in academically oriented classrooms has as its 
purpose the attainment of a level of proficiency to be manifested 
in students' ability to perform on teacher-made or standardized 
achievement tests. On the other hand, it may be that the purpose, 
of learning math in a work setting is to help students become 
functionally competent and able to use subject matter in applying 
it to real-life situations. This would seem to imply that 
students in the workplace actively structure, their subject matter 
to meet the requirements of doing work, while students in a 
traditional classroom, for the most part in a passive role, 
receive the subject matter as contained in textbooks and as 
presented by teachers trained in that discipline. Thus, the 
learning-in-work students may pick and choose subject matter 
content in a way that will enable them to perform work related to 
the goals of the worksite. 

Studying the complexity of students' acquisition of subject 
matter in order that they may become functionally competent 
should include not only what is learned, but also the additional 
variables of what is retained and what content and concepts 
transfer from the classroom to the practical and productive 
performance of meaningful work. As a guide to the investigation 
of the learning and retention of basic skills acquired in the 
workplace, a heuristic model was developed. The purpose of this 
model was to synthesize the complexity of the problem, thereby 
enabling us to conceptualize the variables, to ask questions, and 
to permit the emergence of patterns of cognitive processes and 
possible causal relationships underlying student learning in a 
work environment. 

The model, although neither complete nor comprehensive, 
realistically portrays the sequential nature of lear.ning events _ 

and the availability of data within the pre-established condi- , 

tions of the study. Although the model uses a notation borrowed 
from path-analytic literature, the nature of the data obtained 
for this study (e.g., small sample size, nonrandom assignment of 
students) precludes using techniques such as analysis of 
covariance or estimation of path coefficients. Nevertheless, the 
model, which follows, provides the logical foundation for the 
nature and type of research questions addressed in this study. 
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FIGURE 1 



Heuristic Model for Investigating Student Cognitive 
Processes in Learning Environments 
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Questions 



The purpose was to determine the learning and retention 
patterns of math and reading skills for students in each learning 
environment and to investigate the moderating effects of explana- 
tory variables on those observed patterns* In light of this 
overall purpose^ the following questions were investigated. 

o RETENTION OF BASIC SKILLS 

What is the retention pattern of subject matter 
associated with mathematical skills and reading 
comprehension skills acquired in learning- in-work 
and traditional learning environments? 

o MODERATING RELATIONSHIPS 

Function of Learning Concepts Hierarchically (Bloom's 
Taxonomy) 

Did participation in either the learning-in-work or 
in the traditional learning environment correspond 
to an hierarchical achievement of cognitive 
concepts as theorized by Bloom' s taxonomy and as 
measured by the achievement test? 

Function of Learning Environment (Environmental 
Complexity) 

To what extent did students perceive complexity in 
their learning environment and to what extent did 
these perceptions contribute to math performance? 

Effect of Type of Worksite Experiences 

Did learning- in-work students participation in 
worksite experiences *as classified by the Worker 
Trait Group (from the Dictionary of Occupational 
Titles ) contribute to math achievement? 

Effect of Summer Activity 

Did summer experiences of school or work contribute 
to math achievement at follow-up testing? 

Effect of Cognitive Style 

Did students' cognitive style contribute to math 
achievement? 
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METHODOLOGICAL CONSIDERATIONS 



To provide technical information that meets the various 
needs of the readers , two charts summarize the methodology of the 
study. Readers with varying purposes may choose to read 
thoroughly or to skim those parts of interest to them. Figure 2 
summarizes the conditions for the study including the design and 
student characteristics. Figure 3 displays the measures with 
their corresponding descriptions. The descriptions of the 
measures are organized to correspond with the variables listed in 
figure 1 ( i.e. , independent^ dependent, and explanatory) . 



FIGURE 2 

Methodological Conditions for Conducting the Study 
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DESIGN 

The design for this investigation is probably best described as quasi-experimental, since the 
students were not randomly assigned participate in the EBCE program. Pictorially, the design 
is depicted below: 

Group Events 

Learning-in-Work O-j X-j O2 X3 O3 

Traditional Learning O-j X2 O2 X3 O3 

The design is a repeated-measures one where the nature of the treatments and observations is 
defined as follows: 
Treatment 

X| Learning-in-Work Environment 
X2 Traditional Learning Environment 
X3 Summer Recess 
Observation 

0-| Pre-test ■ September 1 978 (comparability of groups) 

02 Post-test - May 1979 (concepts achieved) 

03 Follow-up test - September 1 979 (concepts retained) 

02 -O-1 9 months (learning interval) 

03 - ©2 3 months (retention interval) 

SUBJECTS . . . . ^ 

Juniors from a suburban school district in Minnesota were selected to participate in tlip study. 
This district was chosen by virtue of having both a traditional learning environment (classroom 
instruction) and learning-in-work environment (EBCE program) in operation. Twenty-seven stu- 
dents who volunteered for the EBCE program were successfully followed over the observation 
period Eleven students in the original EBCE sample of thirty-eight were dropped due to missing 
data at one or more of the subsequent testings. Control students in a traditional environment 
were selected at random from a pool of students to match the EBCE students on sex, school mem- 
bership, and GPA, Twenty-seven control students were successfully followed over the observation 
period, while fifteen students were lost due to missing test data. 

• * 

PROCEDURES 

The Comprehensive Tests of Basic Skills (CTBS) were group administered on three occasions to 
students from both programs who were assembled at their home schools for testing which pro- 
ceeded according to CTBS guidelines. Therefore, testing conditions and times were identical for 
both groups of students for each home high school. Additionally, a Learning Environment Ques- 
tionnaire (LEG) and the Paragraph Completion Test (PCT) were administered at the post-testing 
and interview data was gathered the week prior to the post-testing. A make-up session, following 
the above procedure, was held for each observation to ensure maximum number of respondents. 
Testing time required about an hour. 

GENERALIZABILITY ... 

The fact that random assignment was not used to place students in the learning-m-work program 
places limitations on the extent to which generalizations of the observed results of the EBCE pro- 
gram can be made to other samples .populations of students who would participate in an EBCE 
program Because the students were "self-selected" for membership in the learning-in work pro- 
gram, we do not know the extent to which they are atypical of the "average" high school students. 
Overall it would not be prudent to state that our obtained observations can be expected for a 
group randomly assigned to the program. Essentially, for the learning-in-work group, we can only^ 
describe the way things were for our sample but not predict for a larger population with any sub- 
stantial degree of confidence. 



10 



FIGURES 
Description of the Measures 



INDEPENDENT (PREDICTOR) VARIABLES 

o Student Participation 

Students who participated in the EBCE program were designated as those in the learning-in-work 
environment, while students who participated in classroom instruction were designated as those in 
the traditional learning environment. 

DEPENDENT VARIABLES 

♦ Comprehensive Tests of Basic Skills (CTBS) 

The CTBS Expanded Edition, Level 4, Form S was used at all testings. Of the total test battery. 
Test 2 (Reading Comprehension) and Test 7 (Mathematics Concepts and Applications) were selected 
for administration due to constraints on testing. According to the Test Coordmator^s Handbook 
(1976, p. 5), the CTBS is described as a measuring instrument thus: 

Measurement of the basic skills and abilities cannot be divorced entirely from the measure- 
ment of knowledge acquired through schooling, but it is not the intent of CTBS to measure 
this knowledge directly. The emphasis ... is on measurement of the grasp of broad concepts " 
and abstractions as developed by all curricula and on the facility in the skills that are re- 
quired for effective use of language and number .... 

Of importance to this study is the CTBS classification scheme whereby both content and process 
dimensions are provided. According to CTBS literature, the categories in the content dimension 
(e.g., measurement, sets, problem solving) vary according to the type of content considered appro- 
priate to students of the grades for which it was intended. The categories for the process dimension 
follows essentially the approach presented in Bloom's Taxonomy of Educational Objectives in that 
the classification scheme is hierarchical in nature. Thus, as the level increases, more of the complex 
processes are used with a greater frequency. For the process dimension the categories measure knowl- 
edge of ways and means of dealing with concepts and principles as reflected in the application of rules 
and processes. For the two sub-tests used in this study, the content and process dimensions are as 
follows: 



Dimension 


Mathematics 


Reading Comprehsnsion 




Scale 


Scale 




Name 


Name 


Content 


1. Concepts 






2. Application 


1. Comprehension 


Process/ 
Concepts 


3. Recognition 

4. Translation 

5. Interpretation 

6. Analysis 


2. Recognition 

3. Translation 

4. InterpretEtion 

5. Analysis 



Thus, the CTBS not only measures mathematical and reading comprehension content, but also the 
cognitivo processes or concepts underlying learning. 

EXPLANATORY VAR I ABLES 

* Hiernrchical Order of Concepts 

As described above, the math concepts are the process scales of the CTBS that were developed to 
correspond to an hierarchical order theorized in Bloom's Taxonomy. 
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(Figure 3, continued) 



Learning Environment 

A Learning Environment Questionnaire (LEQ), developed by project staff, was composed of thirteen 
items measuring the degree to which students perceived the environment as providing feedbacic, offer- 
ing a variety of tasks, and giving direction to complete tasks. Factor analysis of the scales resulted in 
three identifiable factors. While the results from the LEQ are discussed later, the factors and their 
corresponding items are presented in figure 4. 



• Worksite Experiencei . 

The worksite experiences of the Learning-in-work Students were classified by the Worker Trait 
Groups of the Dictionary of Occupationai Tities. 

• Summer Activities 

Students were asked to Indicate which activities they were engaged in over the summer. Students 
could choose from summer school, part-time paid work, full-time paid work, or indicate other 
activities. 

• Cognitive Style-Paragraph Completion Test (POT) 

The POT derived from the work of Schroder, Driver, and Streyfert (1967) represents an approach to 
the content-free measurement of cognitive style. The approach Involves a projective technique of 
stem completion. The POT presenU the student with five stems requiring three sentences to be 
written for each stem. The integrative or cognitive complexity score used for this study was the mean 
of the four most abstract responses after each stem had been scored on a seven-point scale. An 
example of a stem used for this study was: 

Your friend: Rulesi How do you f-iel about rules? 

Your answer: Rules * . . (complete iJie sentence and write two more). 
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FIGURE 4 
Learning Environment Questionnaire 



Learning Environment Questionnaire (LEQ)* 
Factor I: Opportunity for generating concepts 

4 In my program i was able to ask many questions about the work, 
8 The work that I did offered me many different things to do. 

10 In my program I was encouraged to come up with my own ideas. 

11 (The adult)* *provided me with opportunities to do meaningful work or solve 
problems. 

13 (The adult)**encouraged me to decide for myself how I was going to do my 
work. 

5 The results of what I did had meaning, I felt the results were important. 



Factor II: Structure that permits learning with adults 

3 (The adult)** taught me what I needed to know. 

6 ^The adults)** described the way they wanted me to do my work. 

9 (The adult)** gave me the right way to do the work. 

12 (The adult)** showed me what was required of me. 



Factor III: Gratification for initiating and carrying out work. 

1 In my program I felt encouraged to find things out for myself. 

2 I was able to tell by myself if I was doing a good job. 

7 In my program I had opportunities to try things out for myself. 



* Responses ranged from 1 -strongly disagree to 5-strongly agree. 

** The learning-in-work students responded to two items, one for the program coordi- 
nator, the other for the resource person. The traditional environment students 
responded to a single item with the adult being the teacher. 
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RESULTS 



This section contains a concise statement of the problem 
followed by a discussion of the results for each research 
question. As far as possible, the results are displayed or 
described in nontechnical ways. We have chosen this approach, in 
part, because the summary is derived from two technical reports 
that carefully detail the research hypotheses, analyses, and 
findings and because the primary focus is on the interpre- 
tations and recommendations which we consider exciting and 
provocative. 

The primary purpose is to examine the retention of mathemat- 
ical and reading concepts for students enrolled in a learning- in- 
work environment (EBCE) and a traditional classroom environment 
on a measure of academic achievement in a twelve-month longitu- 
dinal design. Students' performance (n=27) in each environment 
is evaluated using the CTBS, which was administered at the 
beginning and end of their junior year and at the beginning of 
their senior year. Thus, the learning interval is designated as 
the time between pre- and post-testing, and the retention 
interval, the time between post- and follow-up testing. The 
secondary purpose is to investigate the possible cognitive 
processes and causal patterns among environmental characteris- 
tics, personal characteristics, and performance effects that 
underlie student learning in one of the two learning 
environments. 



Retention of Basic Skills 

What is the retention pattern of subject 
matter associated with mathematical skills 
and reading comprehension skills acquired in 
a learning-in-work and a traditional learning 
environment? 

Since the purpose of the investigation was to identify areas 
for future in-depth examination of the data, repeated-measures 
analysis of variance techniques were used to evaluate the results 
of the math and reading CTBS scores. Since the sample size was 
relatively small, an attempt was made in all analyses to use the 
smallest number of variables possible to allow higher statistical 
power to detect effects that may have bcien present. Thus, anal- 
yses of CTBS math and reading scores were performed using a two 
(programs — learning-in-work and traditional) by three (tests — 
observations at T^, T2/ and T3) design to permit increased 
power in detecting effects. The dependent or response variables 
representing math and reading skills were those measured by the 
six CTBS math scales and the five CTBS rfjading scales. 
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What was the effect of the two environments on standardized 
scholastic achievement performance? Figure 5 displays the 
overall composite form of the relationship across math and 
reading concepts. 

Inspection of the math scores revealed a different pattern 
for the learning- in-work and traditional learning group. Three 
of the six math scales revealed significant interactions of the 
groups by tests, indicating significantly nonparallel learn- 
ing/retention functions. Additionally, the two groups differed 
significantly at the end of the school year but were nonsignif- 
icantly different at the start of their junior year and at the 
start of their senior year- Thus, groups were equivalent at the 
point where they were split into different learning environments , 
but the students in the traditional learning environment 
increased in performance up to the end of the year, while the 
learning- in-work students showed a decrease in performance on the 
post-test- This effect was manifest in two scales at p<.05, 
while three other scales showed trends in this -^direction* The 
groups subsequently reversed this direction of change over the 
summer: the learning of students in the traditional learning 
environment decreased from post- to follow-up test, while that of 
the learning- in-work students increased over the summer. 

Results of reading scores in general, showed both groups 
demonstrating equal performance that increased linearly for four 
reading scale's over the learning interval (pretest to post- 
test) and over the retention interval (post-test to follow-up). 
Significant mean increases were seen only for two reading scales 
when pretest scores were compared to follow-up. The learning- 
in-work and traditional learning groups did not differ 
significantly at any observation, on any reading scale, 
indicating a similar level of performance on these measures. 
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FIGURES 

Composite Functions for Math and Reading SIcitIs 
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Moderating Relationships 



Because of the unanticipated results of math performance, 
the following research questions focus on explaining the 
different math learning and retention functions for the two 
groups of students. 



Function of Learning Concepts 
Hierarchically (Bloom's Taxonomy) 

Did participation in either the learning-in-work or in the 
traditional learning environment correspond to an hierarchi- 
cal achievement of cognitive concepts (at post- or follow-up 
testing) as theorized by Bloom's taxonomy and as measured 
by CTBS? 

The second area of inquiry was aimed at further exploring- 
the unusual patterns of performance exhibited on the mathematics 
scales of the CTBS. As illustrated above, the patterns of 
learning/retention, when broken down by program, suggested that 
different processes were involved for the two different learning 
environments. Thus, did enrollment in one of the two learning 
environments cause' students during the learning or retention 
intervals to learn or retain math skills hierarchically as 
theorized by Bloom's taxonomy and as measured by the CTBS? The 
CTBS process scales of Recognition (R) , Translatipn (T) , 
Interpretation (I), and Analysis (A) were hierarchically 
developed along the lines of Bloom's Taxonomy; therefore, it ' was 
hypothesized that students would learn the concepts in such a way 
that, for example, a mastery of the cognitive skill of 
Recognition at pretest would increase the students' performance 
in the skill of Translation at the post-test. 

One technique for assessing the validity of the above 
hypothesis of different causal processes for each group is 
cross-lagged panel correlation (CLPC). For the purpose of our 
analyses of the CTBS math performance, CLPC techniques were used 
according to the procedures outlined by Kenny (1979). Briefly, 
CLPC is a statistical method that attempts to rule out alter- 
native hypotheses to a causal effect between two variables. The 
primary alternative hypothesis in these cases is that of 
spuriousness , i.e., one variable does not cause the other; 
instead, they are related by virtue of a common relationship to a 
third (unmeasured) variable. For example, the correlation between 
height and math achievement may not be due to a causal relation- 
ship; instead, they may be related because increases in both are 
caused by maturation. 

The usefulness of CLPC lies in its ability to provide 
support for the existence (and form) of a causal relationship 
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between variables by ruling out the alternative hypothesis of 
spuriousness. Therefore, the purpose of these analyses is not to 
"prove" the existence of specific causal relationships in the 
CTBS scales/concepts; rather, it is to discover significant 
differences in the crosslags or nonsymmetric patterns. These 
differences or patterns would be seen as offering support to the 
hypothesis that different causal processes were operative in the 
two learning environments, insofar as the variables represented 
by our observed variables are concerned. Thus, the CLPC 
techniques will be used as suggested by Kenny (1979), in that 
significant crosslag differences will be seen only as indicators 
of a potential causal relationship between the variables (CTBS 
process scales), and not as "proof" of causation.. 

The predicted and actual patterns of learning cognitive 
skills in the two learning environments are depicted in figure 6. 
crosslag differences of p<.10 are reported. 



FIGURE 6 
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These results suggest that for the learning-in-work students 
the cognitive concepts for math were causally related but not in 
the hierarchical order theorized by Bloom's taxonomy as inter- 
preted and "measured by CTBS, That is^ with regard to the 
predictions generated from Bloom's taxonomy, the signs of the 
crosslag differentials (on significant pairs) were not consistent 
with the notion that lower-order concepts should cause higher- 
level concepts. Furthermore, the results would indicate that 
different causal processes were operating in the two environments 
as manifested by the number and type of relationships. The 
learning interval for the learning-in-work students showed the 
greatest number of potential causal relationships. During the 
retention interval both groups showed relationships between two 
variables, but for different cognitive concepts. Overall these 
results are consistent with the hypothesis that there are 
different causal processes operating for learning math concepts 
as a result of enrollment in one of the two learning 
environments. 



Function of Learning Environment 
(Environmental Complexity ) 

To what extent did students perceive complexity 
in their learning environment? 

Because the investigation is concerned with learning and 
retention in two learning environments, a measure to assess 
student perceptions of the complexity of the environment was 
developed. 

Environmental complexity was considered in terms of the two 
types of environments, one in which concepts are generated and 
the other in which concepts are presented. Concept-generating 
environments may include opportunities to observe, ask questions, 
experiment, develop ideas, and evolve alternative problem solu- 
tions. Indicators such as these are characteristic of an open 
environment where individuals are supported and encouraged by the 
environment to achieve goals. On the other hand, concept- 
presenting environments may include those in which learners are 
told what to do, are taught the correct or acceptable way to 
handle a situation, and are given relatively little encouragement 
to be creative in solving problems. Indicators such as these are 
characteristic of a closed environment where individuals may 
perceive the environment as nonsupport ive of achievement of 
individual goals. A concept-generating environment may be 
considered more complex than a concept-giving environment, in 
part because decisions must be made without the benefit of 
readily available rules and predetermined consequences. Thus, 
individuals in such environments may have to consider more 
dimensions and make more judgments with finer discriminations in 
deciding how. to solve problems. 
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A Learning Environment Questionnaire (LEQ) was developed to 
assess the environmental factors discussed above (figure 4). 
initial analysis of the LEQ as a" dependent or outcome measure 
(t- tests between groups) showed that the learning-in-work 
students had significantly higher means on ten of the thirteen 
items (p<.05). These results seemed to indicate that students in 
the learning- in-work environment perceived (1) a greater chance 
to make discoveries on their own (autonomy), (2) more support 
from instructors for developing their own ideas, (3) more 
feedback on their performance (self-generated), (4) more variety 
in their learning tasks, and (5) more meaning in their activi- 
ties. These characteristics are similar to those identified by 
research on learning and retention. Overall they seem to lead to 
increased performance and thereby to favor the learning-m-work 
environment as an effective vehicle for academic learning and 
retention. 

In order to examine the underlying structure of the 
perceptions of the learning environment, as reflected in the 
items of the LEQ, factor-analytic techniques were used. This 
procedure resulted in three identifiable factors. Although the 
LEQ requires additional research for validation, the analysis 
suggests that it does discriminate between student perceptions of 
learning-in-work and traditional learning., environments. The i' 
three factors indicate that students see the learning-m-work 
environment as providing (1) an opportunity for generating 
concepts or engaging in a variety of experiences, (2) a structure 
that permits learning. and negotiating with adults who are respon- 
sible for-guiding the learning, and (3) sufficient gratification 
(feeling of self-confidence) to enable them to initiate and carry 
out work. Environmental complexity is suggested when higher 
levels of the factors are perceived to be present in the environ- 
ment. That is, an environment that is perceived as having more 
opportunities for generating concepts is more complex than one 
that is perceived as having fewer opportunities. When this 
interpretation of the LEQ was applied to the two groups of 
students in the study, the learning-in-work students, when 
compared to the traditional learning students, perceived their 
environment as providing (1) greater opportunity to generate 
concepts, (2) more structure for negotiating work, and (3) more 
gratification for initiating and completing work. Thus, on the 
basis of this measure (LEQ) and the current sample of students, 
the learning- in-work students appeared to perceive a more complex 
environment for learning than the traditional learning students 
perceived. Additional information on the use of the LEQ as a 
predictor of math performance is presented in the final research 
question. 
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Effect of Type of Worksite Experiences 



Did learning-in-work students participation in 
worksite experiences as classified by Worker 
Trait Group (from Dictionary of Occupation 
Titles) contribute to math achievement? 

In the course of their program activities, the students 
in the learning-in-work group chose placements at various 
sites in the community, where their "hands on" experiences took 
place. It was of interest in the present analyses to examine 
whether differences occurred in the math performance of the 
students as a function of the differences in their site 
placements. Specifically/ were there differences between 
students who had at least one exposure to one of the twelve 
worker trait group classified job clusters and those who did not 
have that exposure? If these clusters were valid for the purpose 
of classifying the characteristics of a given site and if some of 
these characteristics had an impact on the acquisition of math 
skills, then it would be expected that there would be differences 
(on post-test and/or follow-up CTBS scores) between participants 
and nonparticipants in those worker trait groups (WTG), differ- 
ences that affected math skill acquisition. While this type of 
analysis could certainly not ad^^ss the issue of what caused 
what (i.e./ Did the WTG classification experience cause later 
math achievement?)/ the discovery of differences as a function of 
placement would tend to contribute support to the EBCE hypothesis 
-that the type of site characteristics has an impact on what is 
learned at the site... In addition to the fact that the students 
self-selected themselves into the sites, the presence of 
differences could also be attributed to an interactive effect of 
the self-selection and the characteristics of the site. This 
alternative explanation, however, could not be addressed in this 
analysis. 

The analyses consisted of using simultaneous regression 
analyses wherein the predictor variables were the WTGs (for those 
students who participated in sites with that WTG classification) 
and the dependent or outcome measures, the two CTBS content 
scales of concepts and application. The results showed differ- 
ences between the participants and nonparticipants at post-test 
for the WTG classifications of 1 (artistic) and 11 (social) on 
the math scale of concepts and for the WTG 11 on the math scale 
of application. While this result would tend to suggest an 
effect for the classifications, the results also showed that 
patterns similar to the post-test differences were present at 
pretest before exposure to the sites. Thus, the presence of 
these initial differences before the experience casts strong 
doubt on the efficacy of the WTG classifications as causal agents 
in this sample. 
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Effect of Summer Activity 



Did sununer experiences of school or work contribute 
to math achievement at follow-up testing? 

The research question addressed by this section concerned 
student activities during the summer break (i.e., between post- 
test and follow-up test) and their relation to math achievement* 
Specifically, these analyses addressed the issue of whether 
distinctions (1) between summer school and no summer school / 
(2) between work and no work, and (3) between learning- in-work 
and traditional programs would account for significant amounts of 
CTBS score variance. 

An alternative explanation of the increase in scores 
following the summer recess could be that this increase was an 
artifact, resulting from greater exposure to learning situations 
(i.e., summer school). If the students had worked during the 
sununer, a similar increase in performance might have been expect- 
ed due to the opportunity to practice skills . learned during the 
school year (provided that such an opportunity existed) . Another 
alternative explanation to significant differences between 
students on the summer-work, and summer-school variables could be 
that these" differences represented pre-existing differences; 
thus, a hypothetical finding that students in summer school 
scored higher at the follow-up test could be due to their 
superiority at pre- and post-test, rather than to the extra 
classwork. - 

To address these issues, simultaneous regression analyses 
were conducted at all testing periods. The effect of summer 
school attendance and a summer job was examined on CTBS math 
performance, while the effect of the program variable was held 
constant (i.e., partialled from each). Summer activities did not 
help much in explaining CTBS score variance, independent of 
program effects; none of the multiple correlations were signif- 
icant at p = .05, and only summer school participation showed 
strong effects (on math content, at post--test and follow-up 
test). The general pattern was that students who went to summer 
school showed lower mean math scores at all testings, while 
students who worked showed generally higher mean scores. Thus, 
summer activity, as measured by this study, does not seem to 
explain the learning- in-work or traditional learning environment 
differences observed for math performance during the retention 
interval. 
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Effect of Cognitive Style 



Did students' cognitive style contribute 
to math performance? 

The position of the authors (and supported by the litera- 
ture) is that the relationship of person, environment, and 
performance (as measured by the CTBS) is interactive. In this 
sample, optimum performance is predicted for students who 
perceive a more complex environment, as indicated by higher LEq" 
scores and who exhibit greater integrative complexity as measured 
by the PCT. Thus, when the level of environmental complexity^ is 
controlled, the performance of students is predicted to be 
uniformly higher as a function of increased integrative complex- 
ity or cognitive style. Accordingly, a significant relationship 
or "main effect" for the cognitive style variable is predicted, 
together with a significant interaction effect when performance 
is predicted from perceived environmental complexity and 
cognitive style. Overall, the hypothesis is that performance is 
maximized when the students ' level of cognitive style is matched 
with similar levels of perceived environmental complexity and 
that performance is minimized when there is a "mismatch" between 
the two variables (e.g., when students with less integrative 
complexity are placed in an environment perceived as complex). 

To assess the predictions regarding the environmental and 
cognitive complexity interaction, hierarchical regression 
analyses were used. While membership in either of the two 
• - - environments- could conceivably have been used as an "objective" 

measure of environmental complexity, this approach was rejected 
at this stage, since it was felt that individual differences in 
perceptions would constitute more meaningful information than the 
"program" variable. The results indicated that no interactive 
relationship was present for the cognitive style and environ- 
mental complexity variables. 

• <• 

In an attempt to discover the person-by-environment inter- 
action through a different conceptualization of the "environment" 
construct, variables representing other aspects of the environ- 
ment were added to the model: (1) summer activities, consisting 
of summer school attendance and employment; and (2) program, 
whether -enrolled" in the learning- in-work or trad^ltional learning 
environment. it was reasoned that the inclusionZ-bf the above 
variables would perhaps provide a more complete representation of 
salient environmental characteristics, and therefore a more 
accurate test 'of the person-by-environment hypothesis. The 
analyses were replicated. The results of using summer activities 
and the program classification were the same as found for using 
the LEQ scales alone in that the main effect variance and the 
interaction of PCT-by-environmental variables produced no signif- 
icant increase in explained variance beyond that explained by the 
main effect variables. 
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in both analyses, however, a main effect or relationship was 
observed for cognitive style as measured by the PCT and for 
perceived environmental complexity as measured by the LEQ. That 
isr higher cognitive style scores were associated with higher 
achievement for both the learning and retention intervals across 
all levels of perceived environmental complexity* Although weak, 
a reverse relationship was generally observed for the perceived 
environmental complexity in that an increase in LEQ scores was 
associated with small decreases in math performance. Overall, 
the cognitive style variable exhibited uniform effects on math 
performance; the environmental perceptions (LEQ) showed a 
smaller, less consistent effect; ^nd the predicted person-by- 
environment interaction failed to materialize. 
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Summary of Findings 



The findings from this study are summarized below. 

Retention of Basic Skills 
o Math Performance : 

During the learning interval / the learning- in-work 
students decreased in performance and the tradi- 
tional learning students increased, while 
during the retention interval, the learning-in-work 
students increased in performance and the 
traditional learning students decreased, 

o Reading Comprehension Performance: 

Overall, both groups demonstrated similar increasing 
performance for the learning and retention intervals. 

Moderating Relationships 

o Function of Learning Math Concepts Hierarchically: 
For the learning-in-work students , the concepts 
learned were not hierarchically related as theorized 
by Bloom's taxonomy and as measured by CTBS, 

Each group showed different causal concept 
patterns for the learning and retention intervals. 

o Function of Learning Environment: 

As measured by the LEQ, the learning-in-work students 
perceived their environment as providing greater 
opportunity to generate concepts, more structure 
for negotiating work, and more gratification that 
enabled them to initiate and complete work than 
the traditional learning students perceived. 

o Effect of Type of Worksite Experiences: 

For learning-in-work students, the decrease in 
math performance during the learning interval 
could not be explained by the different worksite 
experiences as classified by the Worker Trait Groups. 

o Effect of Summer Activity: 

Summer-school attendance or summer work could 
not account for math performance of either 
learning group during the retention interval. 

o Effect of Cognitive Style: 

Students who demonstrated greater cognitive com- 
plexity performed significantly higher on their 
acquisition of math skills for both the learning and 
retention intervals. 
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INTERPRETATION 



The patterns of math performance were not predicted from 
previous research, which would have suggested an increase for 
both groups during the instructional period and a "forgetting" 
gradient over the summer (Ebbinghaus 1964). Alternative post hoc 
explanations for these results were considered by the authors , 
which centered on uncontrolled effects of student selection for 
the learning-in-work program. While it was deemed possible, that 
due to nonrandom assignment of students to the EBCE program, 
subjects may have systematically differed on a variable that 
could interact with the repeated testings, this explanation was 
not favored in view of the lack of differences between groups at 
the start of the study. Additionally, a mechanism would be 
necessary to account for the unequal effect of such a confounded 
selection, such that for math performance it would have no effect 
at test 1, would produce a significant decrement at test 2, and 
would then increase performance for the EBCE students at the same 
time that traditional students (in the same environment) were 
decreasing over the summer. The mechanism would also have to 
take into consideration the generally equal performance of both 
groups at all .three testings for reading performance. This 
necessarily unparsimonious approach was rejected in interpreting 
the results of the math performance. 

Interference and Assimilation Model 

A model was proposed to account for the unequal learning and 
retention of CTBS math skills, a model that emphasized two 
constructs: interference due to previous learning (retroactive 
interference) and assimilation of information. This model is 
displayed in figure 7. The researchers hypothesize that students 
in the learning-in-work environment were learning different 
things than the students in a traditional setting. Specifically, 
it was postulated that EBCE students were learning new "rules" 
for learning how to learn math in . a work environment. These 
rules for learning how to learn were seen to be different than 
those typically measured by academic tests of achievement (e.g., 
CTBS). The paradigm is as follows: (1) students in both groups 
had similar learning histories up to the first testing, which 
emphasized traditional, nonapplied use of math constructs; 
(2) students placed in the learning-in-work environment were then 
forced to use or generate constructs in an applied setting and to 
generalize from abstract math concepts to applied math usage on 
the jobs; (3) traditionally learned math skills were not being 
practiced on the job, and new ways to use math were being learned 
that did not necessarily overlap with previous learning; 
(4) these new rules for learning were not assimilated into the 
previous math framework or structure learned in class, thereby 
constituting a retroactive inhibitor to the retrieval of the CTBS 
measured math skills which emerged as a function of dissimilari- 
ties for the two ways of learning math (measures at T2); and 
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FIGURE? 

Interpretation of Results From A Retroactive Design Perspective 
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(5) follow-up performance on CTBS depended on the extent to which 
new ways to use math were assimilated into the previous framework 
or structure of math knowledge. That is, CTBS performance should 
increase according to how well students could relate the new 
rules for using math to the way math is measured by the CTBS. 
Conversely, it was predicted that the use of a performance-based 
math test ( content- valid for applied settings) would reveal that 
learning-in-work students were not "losing ^information" but were 
learning new rules for learning how to learn, rules which were 
not manifest on the CTBS until they could be assimilated into the 
previously developed traditional structure for remembering math 
concepts. 

, Supporting Evidence 

Other findings can be used as possible supporting evidence 
for the interference hypothesis. For example, while we cannot 
say that the environments caused the observed patterns of cross- 
lag correlations for math concepts, the results do suggest 
(1) that for the learning- in-work group, the causal patterns were 
not hierarchically related, as would be theorized by Bloom's 
taxonomy and (2) that the learning-in-work group demonstrated a 
more diverse pattern of causal relationships during the learning 
interval than the traditional learning group demonstrated. Thus, 
while not providing "proof," this finding does add support that 
some type of interference may have been present for the learning- 
in-work group during the learning interval, as manifested by the 
nonhierarchical nature and the greater number of causally related 
math concepts. 

Another, event that could counter the interference factor and 
explain the decreased math performance of the learning-in-work 
students during the learning interval could be their unequal 
exposure to worksites that, required use and practice of math 
concepts. However, our analyses suggest that exposure to work- 
sites as classified by the WTGs did not affect math performance. 

Also, the interference hypothesis may be supported by the 
students' perceptions of the complexity of the learning environ- 
ments. As measured by the LEQ, the learning-in-work students 
perceived their environments as providing greater opportunity to 
generate concepts, more structure for negotiating work, and more 
gratification that enabled them to initiate and complete work 
than the traditional learning students perceived. Thus, if more 
of these factors are present in the learning-in-work environment 
than in the traditional learning environment (if this is the 
students' first formal exposure to a work experience, they have 
had little or no experience with these factors) then the factors 
may behave as interference-type variables to the acquisition of 
math concepts. That is, these environmental factors in some 
unknown way may interfere with the transfer of math learned in a 
classroom environment to math used or required in a work 
environment. There is some evidence in our sample that this 
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interference occurs, as students who scored high on the LEQ 
generally showed significant statistical decreases ( although 
moderate) on math performance^ Thus, while these factors are not 
unique to the learning-in-work environment, they nevertheless 
seem to be negatively related to — or to interfere with — the 
acquisition of mathematical concepts. 

An event that could counter the assimilation factor and 
explain the increased math performance of the learning--in-work 
students during the retention interval or the decreased math 
performance of the traditional learning students during the 
retention interval could be their summer activities of school or 
work and the resultant practice or nonpract ice on math concepts • 
However, the analyses suggest that summer activities, as measured 
in this study, did not contribute to the math score variance at 
the follow-up testing. 

Limitations of the Model 

It should also be noted that the interference/assimilation 
model is hypothesized only for math concepts and not for reading 
comprehension concepts. It would appear from our sample that 
whatever reading is, it does seem to transfer from prior school 
learning to learning-in-work environments. Whether this is due 
to the structure or content of reading or to the fact that work- 
site mentors require students to be able to read prior to 
worksite placement was not investigated as part of this study. 

While the interference/assimilation model can explain the 
observed math performance patterns, there are, at this stage of 
development of the study, theoretical limitations to the expla- 
nation. As Henry Ellis (1980) points out in his review of the 
first-year results, it is possible that the forgetting occurring 
during the learning interval could be due to other nonspecific 
interference and not to learning specific interference rules or 
concepts . It is also poss ible that there was no interference if 
one assumes that the skills and knowledge learned in a learning- 
in-work environment represent an independent system that neither 
interferes with nor reinforces prior school learning. Thus, 
forgetting may occur because of insufficient practice to maintain 
the level of knowledge exhibited nine months earlier. This 
explanation, however, seems unable to furnish an adequate 
explanation for the increase in performance over the summer when 
there was also no formal practice or learning of math concepts. 

The explanatory construct of assimilation, while intuitively 
appealing , may also possess some 1 imitations as noted by Ellis 
(1980). One is that it may not be required because empirical 
findings show that as time increases for the retention interval, 
retroactive inhibition tends to decrease. Thus , continued 
improvement in performance may be due to the fading of the 
retroactive interference and not to assimilation of information. 
A second limitation centers on the two-process construct itself. 



That is, when, in the learning process, does the interference end 
and assimilation begin? Would we, for example, have found 
interference (or decreased scores) for reading comprehension if 
measurements had been taken more frequently during the learning 
interval. Similarly, would greater decreases in math performance 
have been found earlier if more frequent observations were made? 
For the interference/assimilation construct to be valid, this 
research needs to be replicated with different students and 
criteria measures, and with multiple observations. 

Summary 

In summary, we propose or hypothesize an interf erence/assim- 
ilation model to interpret the patterns of math performance. 
Retroactive interference is used to explain the loss of math 
performance during the instructional or learning interval. It is 
assumed that students in the learning- in-work environment were 
acquiring new skills and concepts that interfered with prior 
learning acquired in a traditional learning environment. During 
the summer retention interval, the learning-in-work students 
assimilated the new rules and concepts learned during the 
instructional interval with their existing knowledge structures. 
This assimilation led to a new cognitive structure which in turn 
led to enhanced performance on the retention test. 



DISCUSSION AND RECOMMENDATIONS 

No longer is formal schooling limited to learning in a 
classroom environment. Not^ only do youth have the opportunity to 
choose their learning environments, but educators concentrate on 
de'^rgning and implementing alternative environments that seem 
.xkely to promote and maximize student learning. The preliminary 
research from this two-year effort has resulted in findings that 
support different patterns of learning and retention of basic 
skills as a result of experience in one of two learning environ- 
ments. At this stage of the research effort, we prefer to 
consider our research exploratory — providing for an initial line 
of inquiry. 

To implement this line of inquiry, this section relates the 
implications of the research findings and interpretations of the 
study to research designs and measurement concerns and -to the 
substance or research variables that should be examined in future 
efforts. Thus, the section addresses two specific areas: 
research design and measurement considerations; and research 
variables related to a study of the learning, forgetting, and 
retention of basic skills in learning-in-work environments. 
Specific recommendations are made regarding the extension of this 
line of inquiry* It should be noted that the discussion draws 
upon the suggestions and ideas generated by the three scholars 
commissioned to review the first-year results. 



Research 



Design 



and Measurement 



Considerations 



Design Considerations 

It goes without saying that the study should be replicated. 
Although confidence in the current findings is limitec^ to a great 
degree by the quasi- experimental nature of the research , 
confidence in similar findings in a replicated study could be 
enhanced by a multivariate research design approach. Reviewers 
of this research have noted that the findings may be due to such 
factors as the Hawthorne effect, the impreciseness and ar:biguity 
inherent in the global nature of the "traditional" and iearning- 
in-work" variables, arid* to the cuing effect of the post- testing 
on follow-up observations. While acknowledging the value of a 
multivariate design to identify precisely and measure both 
learning and retention effects attributable to the independent 
variables and their interactions, we also realize the burdens 
placed on students and the costs associated with implementing 
such a design in a school setting. 

The purpose of exploratory studies of this type is to 
develop research ideas by observing learning phenomena in its 
more or less natural state. In this way major effort can be 
diverted from methodological problems and directed to substantive 
patterns of human behavior. By pressing the methodological 
considerations too far, and too fast, we may inhibit the 
emergence of potential -durable factors associated with learning 
and retention of basic skills. in different environments. Thus, 
it is our opinion and recommendation that further investigation 
is required before a multivariate design is implemented, an 
investigation that would identify significant variables and 
increase understanding of the learning process. 

With this recommendation we propose that on-going evalua- 
tions in school districts be used as the vehicle for data 
collection but that (1) additional data observations and 
(2) additional comparison groups be included. The inclusion of 
more data observations during the learning and retention interval 
should provide a more accurate description of the learning and 
retention functions. The proposed interference/assimilation 
model assumes that at some point interference stops and 
assimilation begins. With additional observations, it may be 
possible to determine more precisely the "true" dimensions of 
these functions. Whereas a decrease in math performance was 
observed for the learning-in-work students, with additional 
observations the interference could be manifested at earlier 
stages, meaning that what was observed was actually part of. the 
assimilation process. Likewise, would earlier observations of 
reading performance have resulted in detection of "interference" 
via decreased scores? More observations would enable us to 
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address such questions. However, since more observations 
increase the risk of interaction between testing and treatment, 
we recommend the inclusion of more comparison groups. In the 
current study, the possible effect of the post-test as a cue for 
the follow-up observation could be isolated if an additional 
comparison group received only the pre- and follow-up tests. 

Another factor related to design considerations is that the 
interpretation of the findings is anchored to the classical 
framework of learning and retention* Most classical studies use 
shorter time frames for the learning and retention intervals with 
precise control over the learning task within experimental 
research designs. In effect for this study, group data and macro 
time intervals were overlayed on the micro cognitive processes 
studied in classical learning and retention paradigms. Thus, 
there is the problem of separating nonschool from school effects 
in the presence of the two types of effects especially in non- 
experimental designs. This problem occurs because our interest 
is in studying how learning and long-term retention processes 
operate in naturally occurring learning environments. However, 
classical learning theory says little about learning and long- 
term retention in natural settings, and experimental designs 
require manipulation of the independent variables as part of the 
treatment. Since the environment we want to study occurs natur- 
ally, the manipulation of the independent variables to accommo- 
date the experimental design would change the treatment under 
study. To some extent, multiple observations would provide the 
frequency of data generally associated with classical learning 
studies, a procedure, that reinforces the need for multiple 
comparison groups. Thus, we recommend that research is needed on 
ways to apply learning theories (classical or contemporary) and 
research designs to study alternative learning environments if we 
are to understand the trade-offs of the learning and retention of 
basic skills. 



Measurement Considerations 

For this study, increased achievement of math and reading 
comprehension skills on a standardized achievement test was the 
learning criterion. As discussed earlier, this criterion is 
appropriate for the traditional classroom learning environment. 
However, the mere lack of increased scores for the learning- 
in-work students may not imply that learning did not occur, but 
rather that participation in the work environment, because of its 
perceived complexity and therefore its potential for interfer- 
ence, could lead to decreased test scores but at the same time 
could produce increases in other outcomes not measured in this 
study. Thus, the criterion of increased cognitive knowledge may 
not be an appropriate learning criterion for a work environment. 
Additionally, the use of a standardized achievement test may not 
be an appropriate outcome measure, either in terms of the content 
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to be learned (since it is curriculum bound) or its psychometric - 
properties. For example, the test items may maximize the 
discrimination among student populations at a given time rather 
than measuring change in performance over a period of time. 

With regard to measurement considerations we propose two 
approaches: (1) the use of tests appropriate to both environ- 
ments and (2) the use of systematic observations by trained 
observers. Although the performance of the learning- in-work 
students was lower at post-test^ it may be that even though they 
possessed the basic knowledge needed to solve the test item> they 
may have had difficulties remembering the needed "rule/* since 
that rule (although taught in the classroom) was not required to 
perform effectively at the workplace. Thus, Xhe use of a 
performance-based math test "(content-valid for applied settings) 
could reveal a reverse in the findings for the two groups (with 
the performance of traditional indicating a relative decrease at 
post-test). We recommend that performance-based math items, 
appropriate for both groups of students, be used in future data 
collections. Furthermore, we recommend that math items from 
National Assessment of Educational Progress (NAEP) be considered 
for initial use, since they were developed for more applied 
situations and provide for interpretations different from those 
obtained from standardized achievement test items. In addition 
to multiple observations of test data, we recommend that 
systematic observations be used to supplement these data. If 
conducted in a rigorous and scientific way, the observations from 
on-site visits cannot only be used to provide accurate descrip- 
tions of the learning process (thus providing initial precision) 
but also can be "triangulated" with, the test data to provide a 
more powerful interpretation of the results. 

Recommendat ions 

In summary, we suggest the following recommendations: 

o Continue the line of inquiry with additional 
exploratory investigations. 

o Employ on-going school evaluations with the inclusion 
of more observations and additional comparison groups. 

o Administer tests (such as NAEP) appropriate to both 
learning environments. 

o Conduct systemmatic observations of the learning 
process at the worksite and in the classroom. 

In line with our original concerns for data collection at 
reasonable costs and within the parameters tolerated by school 
policy, these recommendations would be feasible for most school 
programs . 



Considerations Related to the Study of the 



Learning and Retention of Basic Skills in Work Settings 



Since the process for extending our line of inquiry was 
discussed in the preceding section, the factors related to. the 
learning, forgetting, and retention of basic skills in learning- 
in-work environments is the focus of this section. Earlier in 
the report, we suggested that in order to study the processes 
related to learning in a work environment, it is necessary to 
study the student as both learner and worker, the dynamics of the 
environment, the interaction of the student within the context of 
the. work setting, and the content variables,^ ( in this study math 
and reading skills and concepts). This 'framework provides the 
organizing structure for this section. We will begin with the 
content variables, follow with the dynamics of the learning 
environment, and finally discuss the characteristics of the 
learner. Although the topics cannot be investigated in a linear 
fashion, the sequential order of the topics parallels the 
priority which we would assign to future investigations. 
Although we believe that the interaction of the learner within 
the context of the work setting is probably the most powerful 
predictor of performance, this type of relationship can most 
profitably be addressed statistically only after the associated 
variables have been reliably identified and precisely measured. 

Content Variables 

Evidence from this study suggests, even with methodological 
reservations, that math skills learned in a learning-in-work 
environment are different from those learned in a traditional 
environment and that they may be learned and retained in 
different ways (see figure 5). Furthermore, evidence from the 
crosslag correlations of the math process variables (figure 6) 
suggests that the decrement observed at post-test for the 
learning-in-work students may be due to different causal 
cognitive processes that operate in the two environments. With 
regard to reading comprehension skills the evidence suggests that 
student performance is almost equivaler : in either environment. 

Cognitive or conceptual skills . In addition to the basic 
skill outcomes discussed above, we recommend investigating the 
cognitive or conceptual skills that seem to be present in 
learning- in-work environments. As described by Harold Schroder 
in correspondence with the project staff, these skills include 
(1) the ability to generate conceptual complexity to a level that 
matches the complexity of the environment and (2) the ability to 
use multiple conceptions in information processing. More 
specifically, the first skill entails concept learning, the 
ability to discover concepts others use to make judgments; and 
reality testing, the ability to determine "correctness" of the 
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assumption for making judgments. The second skill requires 
concept transmission, the ability to share one's own concepts 
with others; and concept projection, the ability to project one's 
concepts: to new situations. Based on the interview date from the 
learning-in-work program, it would appear that students in this 
environment have the opportunity to practice these kinds of 
conceptual skills. 

Structure of math . The analysis of math concepts as 
measured by the CTBS suggests that math concepts are related 
differently for students in the two environments. Most notably, 
the math concepts learned by students in the learning-in-work 
environment were not hierarchically related while those math 
concepts learned by students in the traditional environment 
approached a hierarchical order as theorized by Bloom's taxonomy. 
This finding leads us to suggest that participation in a 
learning-in-work environment may shape the students' psycho- 
logical structure of math in such a way that it appears different 
from the logical structure as it is derived from experts and 
conveyed in textbooks. Thus, a hierarchical structure of 
concepts as theorized by Bloom may not be appropriate when 
applied to learning concepts in an actual work setting. Our 
reasoning is as follows: in the traditional environment, we 
would expect a hierarchical order of learned concepts because 
students were presented the logical structure through the 
textbook (concepts in textbooks are presumed to be organized 
hierarchically). In the learning-in-work environment, we would 
not necessarily expect a hierarchical order of learned concepts 
because students had to figure out the structure as a function of 
completing work (concepts encountered in work are presumed not to 
be organized hierarchically). Students in both environments 
would perceive their learning to be meaningful. 

This finding, if replicated, could have implications for 
curriculum development in that disciplines such as math, when 
taught in learning-in-work programs, should be structured to 
assist students to understand the relationships between their own 
psychological structure and the logical structure of the 
discipline. Another possibility is to reorganize math curriculum 
for classroom instruction to reflect the psychological structure 
involved in math to perform work. Such reorganization would be 
designed to increase the efficiency of students in applying the 
basic skill to solving problems encountered in the workplace. We 
recommend that an examination and analysis of the concepts 
related to learning and retaining different subject matter 
disciplines in the workplace be initiated. The goal of such an 
effort would be to understand the relationships between the 
psychological, structure of the students and the logical structure 
of the discipline in order to increase the productivity of 
learners who use and apply math concepts in a work setting. 



Dynamics oF- the Learning Environment 

For us, the workplace as a leaifning environment provides 
exciting and dynamic opportunities for youth. Not only do they 
have the opportunity to learn traditional subject matter, but 
they also are able to participate actively in solving problems 
that result in real consequences, in negotiating work tasks, in 
initiating action to complete tasks, and in practicing social or 
coping skills to relate to a variety of workers. The multipli- 
city of these characteristics highlights the need to classify and 
study learning environments. 

Environmental complexity . To furnish a measure of environ- 
mental complexity for the two environments in this study, the 
. Learning Environment Questionnaire (LEQ) was developed. Analysis 
of this instrument suggests three factors related to student 
perception of the learning environment: (1) an; opportunity for 
generating concepts or engaging in a variety of ! experiences , 
(2) a structure that permits learning and negotiating with 
adults, and (3) gratification (feeling of self-confidence) that 
enables .students to initiate and carry out work. Environmental 
complexity is suggested when high levels of the factors are 
present (or perceived) in the environment. 

Data from this study suggested that the learning-in-work 
students perceived greater environmental complexity (as 
determined by higher test scores) than students in the 
traditional environment perceived. They also suggested that 
student perceptions of environmental complexity affected math 
performance at post- and follow-up testings. Overall, since 
higher levels of perceived environmental complexity resulted in 
slightly lower math performance and since these environmental 
characteristics are present in work environments for this study, 
math achievement (as currently structured and measured) may be 
better acquired in those environments not emphasizing factors 
measured by the LEQ. 

Process of learning . David Ausubel (1980) suggests that the 
process of learning be classified along experiential and didactic 
dimensions. This conceptual framework, presented in figure 8, 
suggests a way to classify learning environments more closely 
related to the cognitive psychologists view of learning. This 
approach illustrated the need to determine the extent to which 
reception or discovery learning actually occurs in either envi- 
ronment. The assumption for this study was that the learning- 
in-work environment emphasized experiential and discovery 
learning, while the traditional environment emphasized didactic 
and reception learning. To validate this assumption, we 
recommend an examination of the learning environments to 
determine the proportions of experiential and didactic learning 
for each. 
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FIGURE 8 

Reception-Discovery and Rote-Meaningful 
Dimensions of Learning 
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DEFINITIONS (Ausubel, 1980, p. 9) 

Reception Learning — the principal content of the material to be learned is presented to 
the learner in a more or less final form and he/she need only integrate it into his/her cog- 
nitive structure for the purposes of retention and transfer to new learning experiences. 

Discovery Learning —the principal content of what is to be learned (i.e., new successful 
problems-solving proposition) must first be discovered by transforming relevant back- 
ground knowledge (previously acquired concepts, facts) in such a way as to constitute a 
means to the end specified in a problem-setting proposition. Once an acceptable problem* 
solving proposition is discovered, it is then internalized in precisely the same way as in 
reception learning. 

Meaningful Learning — (1) that which occurs if the student employs a meaningful learn- 
ing set to his^er existing structure of knowledge and (2) that which occurs if the learning 
task itself is potentially meaningful 
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Variables that effect acquisition and retention of content . 
Variables that produce marked differences in learning may have no 
effect on the rate of forgetting. Benton Underwood in his review 
of the first-year results (1980) pointed out that it is important 
to distinguish between independent variables that influence the 
rate of forgetting and those that do not. Those variables that 
influence the rate of forgetting include the level of learning 
achieved before the retention interval, spaced practice, the 
learning of two or three tasks simultaneously, the learning of 
interfering tasks, and the length of the retention interval. 
Variables that have an effect on learning but not the rate of 
forgetting include meaningf ulness of the task variables, depth- 
of-processing procedures, and individual differences. These 
variables should be considered for future research on learning 
and retention of concepts in alternative learning environments. 
We recommend an analysis of learning-in-work environments to 
determine the presence or absence of these variables. It is 
conceivable that these environments may possess factors that 
contribute more to reducing the rate of forgetting than to 
increasing the differences in learning. 

Initial thinking regarding environmental character istics 
that can affect performance centered on the contrast between 
environments that "generate concepts" and those that "give con- 
cepts." In "concept-generating" environments, according to our 
view, students learn to search for information, generate ideas, 
try them out, and receive feedback. The hypothesis was that 
"concept-generating" environments develop skills that later help 
students acquire knowledge. The results of the LEQ suggest that 
these factors are perceived by learning-in-work students. In 
Henry Ellis' commissioned paper he called attention to the 
generation effect in recall (Slamecka and Graf 1978). The basic 
finding was that if subjects have to generate an answer rather 
than simply to remember one, recall is substantially improved. 
Also> Jacoby (1978) reported that if subjects have to solve a 
problem rather than remember a solution, recall is enhanced. The 
research conducted by Ellis and associates suggests that the 
variability effect, seeing information to be remembered in varied 
fashion, aided recall. As Ellis suggests, (1980, p. 7), these 
findings support the concept of learning-by-discovery and would 
appear to be operative in a learning- in-work environment. To 
some extent the generation effect appears to be similar to our 
notion of "concept-generating" environments. 

Because students in the learning- in-work environment are 
bringing ten years of past school knowledge and learning 
processes to the workplace, the concept of transfer is an 
important issue. While we cannot be certain, one possible 
explanation for the decrement in the math scores of these 
students at post-test may be attributable to interference from 
past school learning. Thus, for students who are making the 
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transition to the workplace as well as for those who are 
experiencing other alternative environments, transfer variables 
not present in the classroom environment may be operating in such 
a way as to interfere with math performance at post-testing. Two 
such variables suggested by Henry Ellis (1980, p. 9) are depth of 
processing and cognitive effort. For example, due to the nature 
of learning activities, students in the learning-in-work environ- 
ment may not process the information they have learned at the 
same depth or at the same intensity as students in the classroom. 
Thus, while increased performance (higher math achievement) is 
important in classroom learning, learning at a workplace may 
require less depth of processing or cognitive effort for student 
success, thereby resulting in lower math achievement at post- 
testing . If such mediating cognitive variables are operating in 
worksite environments but not in other learning environments, 
then they should be identified for studying basic-skill acquisi- 
tion and retention. 



Learner Characteristics 

Individual difference variables must be included in any 
study of learning and retention of basic skills in learning-in- 
work environments. 

For this study, the personal characteristic of cognitive 
style was measured by the Paragraph Completion Test (PCT) , which 
yields an integrative complexity score for the individual's 
ability to generate ideas and to organize ideas or concepts in a 
hierarchical system. Results showed that higher integrative 
complexity scores were associated with higher math achievement at 
post- and follow-up testings for students in both learning envi- 
ronments . 

The major recommendation for studying learner characters is 
that a theoretical template needs to be conceptualized that 
systematically relates individual differences, environmental 
parameters, and criteria measures* Advice from the consultants 
to the project have indicated that the following individual 
difference variables appear to be good candidates: cognitive 
style variables such as cognitive complexity, internal/external 
control, and field dependence; aptitude variables such as general 
intelligence, primary mental abilities, and dimensions of 
cognitive style affecting both reception and discovery learning; 
motivational factors such as intrinsic versus extrinsic, task- 
oriented versus ego-oriented, and cognitive drive ; personality 
variables such as dogmatism, closed- versus open-mindedness , and 
tolerance for ambiguity; and knowledge in content area. 
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Reconunendations 



The above discussion illustrates the complexity of studying 
the f actors^irelated to learning and retention of basic skills in 
work settings. The primary recommendation is that multiple 
research strategies which emphasize interview and observational 
techniques be employed to identify and to describe the factors 
associated with the content variables, environmental variables 
and learner characteristics. For example, a factor such as the 
generation effect has been demonstrated to enhance recall in 
laboratory settings. We propose that this type of factor be 
bperationalized and used as part of an observational schedule to 
detect its presence in learning environments. Furthermore, if 
test data were also available, then factors observed in work 
settings could be used as part of an overall analyses. The 
second recommendation is to conduct content analysis of curric- 
ulum for learning- in-work and- traditional learning environments. 
The analysis should focus on identifying the logical structure of 
subject matter disciplines and the psychological structure 
resulting from participation in work settings. If differences 
exist, then we recommend that curriculum be organized to reflect 
the differences. In this way students should be more productive 
in the workplace when applying subject matter concepts. 
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